1. Field of the Invention
The invention relates to methods and systems for the image analysis, and more particularly, to methods and systems for automatically classifying images into categories.
2. Description of the Background Art
There is currently a growing demand for computer systems that can produce, edit, and manipulate graphic images, and in particular, for systems that can create, edit, or manipulate digitized video images. This demand is generated by several market influences and consumer trends. There has been, and will continue to be, steady growth in the multimedia marketplace for computer-based products that integrate text, audio, graphics and increasingly video, for educational, entertainment, and business purposes. Also, the use of video for educational or business presentations and for artistic or personal applications has become increasingly popular as the costs of video production equipment has fallen. Products ranging from video games to computerized encyclopedias to computerized training guides now commonly employ digitized video to entertain, educate, and instruct.
These consumer trends are matched by various technological advances that have made widespread the use of video for computer based applications. Equipment to digitize video at high speed and quality has allowed software designers to integrate video into commercial software products such as computer games, and has allowed individual computer users to incorporate video into business presentations or other similar projects. Improvements in telecommunications and network technologies, such as increased transfer rates, bandwidth, and the like, have made realistic the opportunity for computer users of all types to access online libraries of video with acceptable speed and quality.
The rise of desktop video production, including the development of video compression standards such as MPEG, have reduced the cost of video production systems, making pre- and post- production systems accessible to more users and businesses. There are now available a number of software products for multimedia authoring that handle video, graphics, audio, animation in the development environment. Such technologies have been made possible by increases in microprocessor power coupled with dramatic reductions in cost. Personal computers now offer performance previously found only in engineering workstations, or mainframes.
In addition to computation power and sophisticated software, improvements in storage capacities and compression technologies have increased the ability to store digitized video, which typically requires large storage needs. Uncompressed NTSC quality video requires 15 Mb per second for 30 fps video, or almost 1 Gb for a minute's worth of video. The MPEG standard for video image compression provides for a 40:1 compression ratio, allowing a hour's video footage in about 1.3 Gb of storage capacity. Compression also facilitates network access, and thus the developments of video libraries that allow user to select and retrieve video footage in real, or near real time.
All these factors have produced a demand for systems and products that aid the storage, identification, and retrieval of graphic images and video. This is because designers of multimedia software products, computer graphic artists, and even individual users, often have extensive libraries of digitized photographs, digitized video, or other computer generated graphic images, for incorporating such materials in multimedia products. Thus a designer may have hundreds, or thousands, of images of people, animals, urban settings, landscapes, sporting events, or any other category of images, and may have hours of similarly diverse video footage, all useful for creating multimedia presentations. Similarly, with the emergence of desktop video production, video producers will typically develop extensive libraries of video for use by themselves, or others, to aid in the creation of new works. Other businesses that have existing libraries of video, and that generate large quantities of video, such as television stations, film studios, and the like, will eventually produce and store increasing quantities of video using computers and mass storage devices.
To effectively use a library of images or video, the software designer must be able to retrieve an image or video according to certain visual attributes or characteristics in the image. For example, the designer may need an single image or even video footage of a sunset over a ocean shore for a given project, and would need a way to locate that image from many other images, without having to review many hours of video, or numerous photographs that may or may not match the desired visual characteristics of the image. In the past, such retrieval was manually performed. For computer based image retrieval to be useful, some type of image analysis and classification of the visual characteristics of the images is necessary in order to speed up the retrieval process and make computer based storage an effective and efficient tool.
The visual attributes or statistical qualities of images have been extensively researched, and there are many techniques for determining various aspects of an image, such as density and distribution of its colors, the presence and degree of motion between two images, the presence and position of distinct objects, and the like. However, most of these techniques have been developed for use in two principal areas, compression techniques for communicating or storing images and video, and pattern recognition techniques for determining whether a particular image matches a given reference, such in industrial part inspection.
These various image analysis techniques have not previously been used for classifying images. Rather, classifying images is typically based on storing images in a database with descriptive text annotations. The designer then searches by inputting a text description of an image and attempting to locate images that have a matching text description. There are numerous problems with using this approach to classify images and video.
First, a human observer must view each image in the database. This is an extremely time consuming process, especially in a database that may contain thousands of images, and must be repeated for each image added to the database. Second, during viewing, the observer must decide which visual elements of an image are significant in determining the proper classification of the image. This subjective judgment may overlook various image details that may later be part of image characteristics for which the user is searching by reviewing a list of classification. Thus the observer may not note or descriptive specific objects in the background of image, or implicit elements of an image or video such as panning or zooming. Even in still images, the user may overlook significant colors, shapes, the presence of persons, or other elements. As a result of these subjective judgments, the observer's classification of the image may be either too general (classifying an image of a sunset over the beach as merely a "Sun & Sky") or too specific ("Sunset on The Strand"). When the classification is too general, many dissimilar images will be included in the classification, thereby diluting the value of the classification for discriminating images. Where the classification is too narrow, too few images will be included in later classifications, thus increasing the number of distinct classifications that the user must review in order to locate a desirable image.
Classifying video even more difficult and time consuming. In order to classify a video, an observer must view the entire video, noting its various contents, such as different scenes, and when each occurs, along with a description of each scene and aspects significant for later retrieval. Again, not every feature will be noted by the observer; this is an even more significant problem for video since there is typically more "content" to a video in terms of varying images than a single photograph, and thus a single classification of video is likely be inadequately descriptive of all of the content. None of these approaches use computer based analysis of the images to classify a desired image.
Pattern recognition techniques have been previously used to classify images with computers. These techniques have usually been specialized to a particular field, for example analysis of satellite imagery or component identification for defect analysis. In addition, these techniques have typically dealt only with still images not video. Existing techniques have generally hardwired the classification engine since only a small number of known categories is typically of interest. However, for general video analysis it is necessary to provide more flexible classification methods and to allow inclusion of time based features such as motion.
Accordingly, it is desirable to provide various methods for classifying images according to their image attributes for later retrieval. Where a user creates numerous images or is constantly adding such images to an image database, automatic classification of images should categorize new images on the basis of various user supplied criteria. In addition, it is desirable to provide for adaptive learning of the user's classification of images based on image attributes in user classified images.